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ASSESSMENT OF CONTENT UNDERSTANDING THROUGH 
SCIENCE EXPLANATION TASKS 

Christy Kim Boscardin, Barbara Jones, Claire Nishimura, 

Shannon Madsen, and Jae-Eun Park 
CRESST/University of California, Los Angeles 

Abstract 

Our recent review of content assessments revealed that language expectations and 
proficiencies are often implicitly embedded within the assessment criteria. Based on a 
review of performance assessments used in high school biology settings, we have found a 
recurring discrepancy between assessment scoring criteria and performance expectations. 
Without explicit scoring criteria to evaluate the language performance, it is difficult to 
determine how much of the overall performance quality can be attributed to language 
skills versus content knowledge. This is an especially important validity question for 
English Learners (ELs) under the current state assessment mandates. To date, studies of 
the validity and consequences of standards-based assessments for ELs have been limited. 
In the current study, we examined the various cognitive demands including language 
skills associated with successful performance on content assessments. Also, as part of the 
validity investigation, we developed and examined the relative sensitivity of 
performance-based assessment, which is constructed to be a more proximal measure of 
student understanding and sensitive to detecting instructional differences. 



1 



CHAPTER 1: INTRODUCTION 



Background 

Inclusion of English Learners (ELs) in high-stakes tests raises questions about the 
validity and reliability of these tests to inform decisions about accountability and instruction 
(August & Hakuta, 1997). Under the No Child Left Behind (2002) mandates, by the 2007- 
2008 academic year, all states are required to administer science content assessments to ah 
students, including ELs. However, the Department of Education has not yet provided states 
and schools with adequate technical assistance and resources necessary to develop valid 
content-based assessments. To date, studies of the validity and consequences of standards- 
based assessments for ELs have been limited in number. 

One concern stemming from the inclusion of ELs in high-stakes tests is the question of 
the validity of content assessments. Are content-based assessments actually measuring 
students’ content knowledge and skills, or are these tests unintentionally assessing students’ 
language proficiency? Without a clear determination of the validity of the standards-based 
assessments used in high-stakes testing for ELs, it will be difficult to evaluate the effect of 
the recent standards-based reform initiatives. One of the presumptive benefits of standards- 
based performance assessments is that they can more effectively measure complex thinking 
and problem solving. Measuring these skills may encourage schools to address standards 
related to thinking processes instead of focusing on the kinds of drill-and-kill basic skills that 
have often been associated with high-stakes multiple-choice tests. However, the validity of 
these types of measures for ELs is still in question. Another issue concerning the inclusion of 
ELs in high-stakes testing is the persistent achievement gap between ELs and non-ELs 
(Abedi, 2004; Aguirre-Munoz et al., 2005). As states struggle to provide and achieve high 
standards for all students as well as demonstrate yearly progress towards meeting these goals, 
understanding the factors attributed to performance gaps will be paramount. In addition, our 
review of state content assessments revealed that language expectations and proficiencies are 
often implicitly embedded within the assessment criteria. Currently, 27 of the states require 
either an extensive or short written constructed responses as part of their statewide science 
assessment. Given the implicit language demands required by these assessments, without 
explicit scoring criteria to evaluate the language performance, it is difficult to determine how 
much of the overall performance quality can be attributed to language skills versus content 
knowledge. 

In addition, previous studies have shown that the proximity or the degree of alignment 
of assessments to curriculum content also impacts the evaluation of instructional sensitivity 
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(Aguirre-Munoz et al., 2005; Ruiz-Primo, Shavelson, Hamilton, & Klein, 2002). According 
to Popham (2006), instructional sensitivity of test refers to the degree to which students’ 
performances on that test accurately reflect the quality of instruction that was provided to 
promote students’ mastery of the skills assessed by the test. As part of the validity check, 
assessment must be able to differentiate the impact of various instructional qualities. This 
type of validity evidence will be paramount for the current accountability system that relies 
significantly on test scores to make judgments about instructional and educational quality. 

For the current study, we propose to examine the various cognitive demands including 
language skills associated with successful performance on content assessments. Also, as part 
of the validity investigation, we will examine the relative sensitivity of performance-based 
assessment which is purported to be a more proximal measure of student understanding 
compared to standardized assessment such as the California Standards Test which represents 
a more distal measure on detecting instructional differences. 

Content Assessment and ELs 

There has been growing concern over the validity of content assessments with EL 
students, as previous research studies have shown that students’ performance on content area 
assessments is confounded with language proficiency (Abedi & Leon, 1999; Bailey, 2000; 
Butler & Castellon- Wellington, 2000). Bailey suggests, based on a linguistic analysis of 
current standardized tests in content areas, that the language demand of a content assessment 
is a potential threat to the validity of the assessment and may not provide an accurate picture 
of EL students’ content knowledge. As acknowledged in the Standards for Educational and 
Psychological Testing (American Educational Research Association, American 
Psychological Association, & National Council on Measurement in Education, 1999), the 
language demand of content assessments may introduce construct-irrelevant components into 
the testing process for EL students. Furthermore, it was concluded that these test results may 
not accurately reflect the competencies which are intended to be measured. Additionally, 
previous research suggests that most content assessments conducted in English inadvertently 
function as language proficiency tests rather than assessments of content knowledge (Abedi, 
Leon, & Mirocha, 2003). 

To help minimize the effect of construct-irrelevant factors on the assessment of EL 
students, accommodations have been suggested and used in large-scale state assessments. 
However, sources of evidence for validity and effectiveness of these accommodations in 
reducing language factors in content assessments have been limited (Abedi, Courtney, & 
Leon, 2003). Also, certain types of accommodations are irrelevant and difficult to implement 
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in the context of performance assessments. For example, allowing extra time and use of a 
dictionary may help reduce cognitive and linguistic loads in a standardized testing format 
(i.e., multiple-choice tests); however, for performance assessments that typically include 
reading and writing components, extra time and the availability of a glossary are part of 
general assessment guidelines and may not provide additional accommodations for ELs. 

Additionally, a problem may arise when rating the quality of student work on 
performance assessments if the evaluation criteria do not distinguish the language demands 
of the task from the content demands, making it difficult to determine how much of each of 
these factors contribute to the overall score and impression of student performance. 
McCutchen (1986) posits that students’ linguistic sophistication in writing (i.e., ability to 
demonstrate control over the linguistic resources needed to realize a particular genre) can 
compensate for a lack of content knowledge, but high levels of content knowledge do not 
necessarily compensate for a lack of writing proficiency. A high level of content knowledge 
certainly does not, in itself, work to the detriment of student writing. If, however, a writer 
does not understand how texts are constructed or how to use language in a text to lead a 
reader through an argument or an explanation of a process, then the value of the writer’s 
content knowledge may be weakened because the writer cannot communicate this knowledge 
effectively to others. This notion is particularly relevant and potentially problematic for ELs 
when the assessment format requires responses in writing. 

Our evaluation of commonly used performance assessments also reveals that language 
expectations are often implicitly embedded within the assessment criteria. Based on a review 
of performance assessments used in high school biology settings, we have found a recurring 
discrepancy between assessment scoring criteria and performance expectations. For example, 
in the Advanced Placement (AP) Biology exam scoring guidelines, points are awarded to 
student writing based on the inclusion of certain content information. The final score is the 
accumulation of these points, ffowever, we found that the student prompts for the AP 
Biology writing exam ask students to describe, discuss, and explain the biology content, 
whereas the scoring criteria do not explicitly take into account students’ varied ability to 
undertake these communicative writing processes which demand specific academic language 
competence. For example, to write a clear description of a biology concept, students need to 
include specific content vocabulary that is condensed into expanded noun phrases (a noun 
plus any modifying information, such as “the vast number of cells needed for the embryo to 
take form”). Thus, it is unclear whether the raters’ scores clearly measure content 
understanding or a combination of content understanding confounded with the students’ 
language proficiency for describing and explaining. 



4 



Current biology textbooks also frequently include writing components as part of the 
regular chapter assessments. For example, in the 2006 Prentice Hall Biology textbook, 
(Miller & Levine, 2006) students are asked to complete writing assessments called “Writing 
in Science” as part of the end-of-chapter assessments. This task entails writing a paragraph or 
group of paragraphs on target biology content. Like the AP Biology writing exam, Prentice 
Hall’s writing assessment criteria explicitly refer only to the scoring of content knowledge. 
For example, in response to a prompt in which students are asked to write a paragraph that 
includes: (a) an explanation of a polymer, (b) a description of organic compounds, and (c) 
how these organic compounds are used in the human body, the evaluation criteria relate only 
to biology content (e.g., one of the criteria requires that students “explain that a polymer is a 
macromolecule made up of monomers joined together”). None of the evaluation criteria 
pertains specifically to the language features needed to successfully provide an explanation 
of the content. As with the AP Biology Exam, students are expected to communicate science 
concepts through specific academic language processes, though these processes are only 
implicitly evaluated as part of the assessment score. Without explicit scoring criteria to 
evaluate language, it is difficult to determine how much of the performance quality can be 
attributed to language skills versus content knowledge. 

An additional measurement concern associated with typical writing assessments is the 
use of holistic rubrics to rate student writing and content knowledge together. Holistic rubrics 
usually include a variety of criteria (e.g., content, organization, spelling, grammar, etc.) and 
result in a single, overall score of the writing quality. An example of this is the holistic rubric 
used to score the Scholastic Aptitude Test (SAT) writing assessment, which rates student 
writing in consideration of five different points. Roughly, these five points are: (a) 
development of a point of view with clear evidence and use of critical thinking; (b) focus, 
smooth progression of ideas, and good organization; (c) skillful use of language with varied, 
accurate, and apt vocabulary; (d) meaningful sentence structure variety; and (e) correct 
grammar, usage, and mechanics. This type of rubric allows raters to rate the overall effect of 
the writing and its success in communicating ideas to the reader. 

The rubrics used to score the SAT allow a rater to explicitly consider several aspects of 
student writing that contribute to its quality. In cases where a student does not receive a high 
score, however, a holistic rubric does not provide enough information to determine whether 
the student was lacking in academic language proficiency or in their understanding of the 
target concepts. Our rationale and conceptual framework for a new approach to assessing 
content understanding using an already widely accepted and utilized performance assessment 
format are discussed next. 
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New Assessment Framework 



For this study, it was desirable to choose a context in which we could assess students’ 
academic language proficiency as well as specialized content knowledge in the format of a 
performance assessment. To this end, we chose high school biology classrooms as the 
context for this study. Fligh school biology classrooms are appropriate for this endeavor 
because they tend to rely on science texts as a resource for gaining and expressing science 
knowledge. 

As indicated by the National Science Education Standards (National Research Council, 
1996), students need opportunities to present their understanding and to use knowledge and 
the language of science to communicate scientific explanations and ideas. Grounded in the 
National Science Education Standards, we define acquisition of conceptual understanding in 
science to include not only the mastery of particular concepts and ideas but also the ability to 
effectively integrate concepts into the formulation of persuasive explanations for particular 
problems or phenomena. The National Center for Research on Evaluation, Standards, and 
Student Testing’s (CRESST) model-based assessment uses standard architectures embedded 
in disciplinary content to assess core types of learning — basic knowledge, conceptual 
understanding, problem solving, communication, and teamwork. Two different templates 
have been developed and extensively validated as a way to observe science content 
understanding and communication, one using writing explanations given primary source 
materials, and the other using computer-based knowledge mapping to display comprehension 
(Baker, 1994; Chung, O’Neil, & Fieri, 1999). As described below, CRESST’s explanation 
architecture for assessing basic content knowledge and conceptual understanding provides a 
primary framework for the proposed assessment in the current study. 

The explanation architecture seems particularly suitable for the current study context in 
that science is about the construction of theories that explain how the world operates, and 
scholars have noted that discourse, explanation, and argumentation are at the heart of science 
learning (Boulter & Gilbert, 1995; Duschl & Osborne, 2002; Erduran, Simon, & Osborne, 
2004; Pontecorvo, 1987). The explanation architecture provides a format for examining 
whether students are able to integrate a complex structure of biological concepts, including 
related concepts, relationships between concepts, reasons for these relationships, and ways to 
explain and predict other natural phenomena. Within school-based science writing, 
explanation tasks are a dominant genre of student writing (Martin & Miller, 1988) and lend 
themselves easily to on-demand assessment conditions. Moreover, science explanation, as a 
type of formal technical writing, is important in the development of skills that logically 
organize scientific and technical knowledge into explanatory strings which utilize an 
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appropriate science register. 1 When students are provided with increased opportunities to 
practice the genre of explanatory writing, they gain competence in assembling explanations 
that use a proper science register and can move toward functioning as more confident 
participants in the scientific and technical subject areas. 

In the proposed assessment model, the use of an explanation task with reading prompts 
and a writing component provides us with an integrated approach to measuring students’ 
ability to understand and communicate their science knowledge, using appropriate literacy 
skills to convey that knowledge. Under this model, learning science also becomes learning to 
use the specific linguistic features and structures that help communicate scientific principles, 
knowledge, and ideas (Fang, 2004). 

Using this proposed assessment model, in this study we are interested in: (a) accurately 
measuring ELs’ content knowledge in performance assessments, (b) gaining a better 
understanding of whether this measurement of content knowledge is confounded with 
measurements of language proficiency (i.e., the ability to write using the target science 
register), and (c) exploring how these measurements relate to impressions of overall essay 
quality. Based on these goals, we developed three separate rubrics to evaluate student 
responses. The three rubrics include: (1) the holistic or overall quality of student writing, (2) 
student understanding of the target science content, and (3) student ability to express 
scientific ideas in the target register using appropriate academic language. Each student 
essay, then, received three separate scores: Holistic, Content, and Language. 

Rating student writing based on three separate rubrics allows us to determine whether, 
in general, strengths or weaknesses in student writing are more closely related to biology 
content understanding or academic language proficiency. Secondly, scoring in this manner 
allows us to discern whether raters’ impressions of the overall quality of student writing is 
more closely aligned with students’ content understanding or with their use of language. This 
will ultimately shed light on whether rating systems based on content, such as the AP 
Biology exam, or on a holistic system, such as the SAT writing exam, are valid measures for 
assessing EL content understanding and whether an explicit rating of language is needed in 
order to accurately measure written performance assessments. 

Development of the Integrated Learning Assessment 

The assessment we developed is called the Integrated Learning Assessment (ILA). For 
the purposes of research and validation, we developed four versions of the ILA, two in 

1 Register refers to “the configuration of lexical and grammatical resources which realizes a particular set of 
meanings” (Schleppegrell, 2004, pp. 45-46) 
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genetics (“Genetics 1” and “Genetics 2”) and two in physiology (“Physiology 1” and 
“Physiology 2”). Each contains a reading passage, questions on students’ reading processes, 
multiple-choice reading comprehension questions, and a writing prompt. The theoretical 
framework for the development of the ILA is described next. 

As part of the development work, California state content standards for Biology/Life 
Sciences were reviewed extensively. The content standards are designed to define the 
knowledge, concepts, and skills that students are expected to acquire at each grade level or 
grade-level span. The concept of biology, which informs the standards, is described in the 
California Department of Education’s (CDE) Science Curriculum Framework. The CDE 
conceptualization is consistent with generally accepted notions of biology. The field of 
biology is conceptualized as the study of life. The standards describing the field of biology 
focus on the common qualities of various life forms and the idea of the organism. In the CDE 
framework, the study of biology is organized into sections consistent with sub-fields of 
scientific study, such as ecology, physiology, and evolution. 

Ontology 

As part of the development process, after studying the biology standards, we created an 
ontology — a systematic arrangement and categorization of concepts in a field of discourse. 
Developing this type of organizational structure allowed us to uncover the relationships 
between different biology concepts (e.g., what concepts encompass the precursor knowledge 
set needed to understand a specific standard). This involved unpacking and elaborating the 
standards to create a hierarchy of conceptual infonnation. This hierarchy of information was 
then used to: (a) create a framework for content understanding, (b) shape the design of the 
ILA, and (c) guide the development of the content rubric. Using the California Science 
Teachers Association’s Making Connections: A Guide to Implementing Science Standards 
(Bertrand, DiRanna, & Janulaw, 1999) as a guide, we examined the standards in two specific 
science content areas — genetics and physiology. We determined what prior content 
knowledge would be necessary to understand the target standards, based on preceding units, 
content standards and sub-standards, and science standards from preceding grade levels (e.g., 
California Science Standards for Grade 7). 

The two sections of Biology targeted for the ILA are genetics and physiology, 2 both of 
which are well represented in the California Science Test (CST) for biology. We chose these 
two topics for the ILA because we expected teachers to spend more time covering these 
content areas during classroom instruction, permitting us to relate opportunity-to-leam (OTL) 

2 see http://www.cde.ca.gov/be/st/ss/scbiology.asp 



variables with student performance on science achievement tests. Overall, the review of 
content standards and the development of a biology ontology provided us with the context for 
determining the specific content targeted in the ILA. 

Approach to Language Evaluation 

In order to develop the various language-based assessment tasks in the ILA, reading 
and writing standards were examined, as well as concepts from systemic functional 
linguistics. The English Language Arts standards represent the primary guide for this 
development, whereas the systemic functional linguistic approach to language provides 
supplementary information to define distinctive features of academic language in science, 
which are different from those of informal spoken language. Using this framework, we were 
able to isolate and evaluate the specific linguistic features that are critical for proficiency in 
scientific literacy. 3 

In order for students to become scientifically literate, they must be able to read and 
comprehend scientific texts; this includes understanding the highly specialized academic 
language of these texts. They also need to be able to employ these same academic language 
resources to communicate what they have learned. Writing in science is realized through the 
appropriate use of a distinct set of lexicogrammatical resources (Fang, 2004). Using the 
functional linguistic approach to the analysis of academic language in science, we can 
systematically evaluate the effective use and control of the unique linguistic forms and 
structures, which are specific to communicating scientific knowledge. 

The functional linguistic approach has also been applied in the past to analyzing 
students’ writing performance, such as description (Schleppegrell, 1998, 2003), narrative 
(Christie, 1986), scientific essay (Christie, 1986; Schleppegrell, 2003), literary analysis, 
opinionated text (Christie, 1986, 2002), and response to literature (Aguirre-Munoz et al., 
2005). Functional linguistic analyses of student writing reported in previous studies reveal 
that students often lack understanding of expected language use in performing given 
academic tasks. For example, in a description task, students often invoke a non-academic 
interpersonal context (i.e., a situated, personal context) by employing the past progressive 
tense and first person references (Schleppegrell, 1998). These qualities indicate students’ 
difficulties in producing the linguistic features, which are within the realm of academic 
writing. 



3 In keeping with the California Content Standards in Language Arts, an explanation may be alternately referred 
to here as an essay. 
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Description of ILA Development 

Guided by these two approaches to language and content testing, our particular test 
specifications are closely aligned with the criterion-referenced measurement (CRM) 
paradigm. The purpose of utilizing this mode of test development was to have a measure that 
was independent of reference to the performance of other students. 

The test specifications followed a precise format for all four versions of the assessment 
(Davidson & Lynch, 2002). Table 1 gives a brief description of this format. This format is 
discussed in more detail in the following sections (see Appendix A for the ILA test 
specifications). 

Table 1 



Test Specification Format 



Specification 


Description 


General description 


The general description is a detailed description of what is to be tested. 


Prompt attribute 


The prompt attribute is a detailed description of what is to be given to the test 
takers and what they will need to do. This includes the test format (multiple 
choice, essay, short answer). 


Response attribute 


The response attribute, though somewhat overlapping with the prompt 
attribute, is a description of what should occur when the test takers are given 
the prompt attribute. 


Sample item 


The sample item is an example of the specific format and content patterns for 
test items that are written from the test spec. 



The first phase in developing the test specification for the ILA was to provide a detailed 
description of what was to be tested. It was determined, based on the review of the standards 
that the ILA should incorporate high-level cognitive skills such as analysis, elaboration, 
evaluation, and synthesis of ideas learned in biology class combined with the ideas presented 
in the ILA reading passages. The tasks in the ILA were aimed at eliciting students’ use of 
higher-level cognitive skills when engaging in reading and responding to the text through 
writing. 

A specialized general description (GD) was constructed for each of the four versions of 
the ILA. Each version of the ILA included unique biology content standards to be evaluated, 
though the reading and writing standards were the same for all versions. The standards on 
which the GDs were based can be viewed in Figures 1- 3 below. 
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Biology/Life Sciences Standards 



ILA: Genetics 1 

Standard 5: The genetic composition of cells can be altered by incorporation of 
exogenous DNA into the cells. 

Sample basis for understanding this concept: 

5a. Students know the general structures and functions of DNA, RNA, 
and protein. 

5b. Students know how genetic engineering (biotechnology) is used to produce 
novel biomedical and agricultural products. 

I LA: Genetics 2 

Standard 4: Genes are a set of instructions encoded in the DNA sequence of each 
organism that specify the sequence of amino acids in proteins. 

Sample basis for understanding this concept: 

4a. Students know the general pathway by which ribosomes synthesize proteins. 

4d. Students know specialization of cells in multicellular organisms is usually due 
to different patterns of gene expression. 

iLA: Physiology 1 

Standard 10: Organisms have a variety of mechanisms to combat disease. 

Sample basis for understanding this concept: 

10a. Students know the role of the skin in providinq nonspecific defenses aqainst 
infection. 

10b. Students know the role of antibodies in the body’s response to infection. 

ILA: Physiology 2 

Standard 9: As a result of the coordinated structures and functions of organ systems, 
the internal environment of the human body remains relatively stable (homeostatic) 
despite changes in the outside environment. 

Sample basis for understanding this concept: 

9b. Students know how the nervous system mediates communication between 
different parts of the body and the body’s interactions with the environment. 

9c. Students know how feedback loops in the nervous and endocrine systems 
regulate conditions in the body. 



Figure 1. Biology/life standards targeted in the ILA 
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ILA: (All) 

Reading Comprehension Standards: 

2.0 Read and understand grade-level appropriate material. Analyze the organizational 
patterns, arguments, and positions advanced. 

Structural Features of Informational Materials 

2.5 Extend ideas presented in primary or secondary sources through original 
analysis, evaluation, and elaboration. 

Figure 2. Reading comprehension standards targeted in ILA 
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ILA: (All) - Writing Standards 



Writing Strategies: 

1 .0 Write coherent and focused essays that convey a well-defined perspective and 
tightly reasoned argument. The writing demonstrates students’ awareness of the 
audience and purpose. 

1.1 Establish a controlling impression or coherent thesis that conveys a clear and 
distinctive perspective on the subject and maintain a consistent tone and focus 
throughout the piece of writing. 

1.2 Use precise language, action verbs, sensory details, appropriate modifiers, and the 
active rather than the passive voice. 

Writing Applications: 

2.0 Combine the rhetorical strategies of narration, exposition, persuasion, and 
description to produce texts of at least 1 ,500 words each. Student writing 
demonstrates a command of standard American English and the research, 
organizational, and drafting strategies. 

2.3 Write expository compositions, including analytical essays and research reports 

a. Marshal evidence in support of a thesis and related claims, including 
information on all relevant perspectives. 

b. Convey information and ideas from primary and secondary sources accurately 
and coherently. 

c. Anticipate and address readers’ potential misunderstandings, biases, and 
expectations. 

d. Use technical terms and notations accurately. 

Written and Oral English Language Conventions: 

1 .0 Write and speak with a command of standard English conventions. 

Grammar and Mechanics of Writing 

1.1 Identify and correctly use clauses (e.g., main and subordinate), phrases (e.g., 
gerund, infinitive, and participial), and mechanics of punctuation (e.g., semicolons, 
colons, ellipses, hyphens). 

1.2 Understand sentence construction (e.g., parallel structure, subordination, proper 
placement of modifiers) and proper English usage (e.g., consistency of verb tenses) 

1 .3 Demonstrate an understanding of proper English usage and control of grammar, 
paragraph and sentence structure, diction, and syntax. 



Figure 3. Writing standards targeted in the ILA 

Prompt Attribute: Text Selection 

For the purpose of choosing the reading passages for the ILA, it was necessary to have 
a clear understanding of what linguistic resources are used to create scientific meaning and 
what level of reading comprehension proficiency is required at the high school level. To gain 
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